Augmented reality devices display graphical images superimposed over a user's view of a physical environment. The graphical images may be world locked to appear as if they are affixed to an object in the physical world. To accurately position these graphical images so as to appear world locked, an augmented reality device continually estimates its pose (position and orientation) within the physical environment. One source of data that the augmented reality device may use to estimate its pose is optical feature data in captured images of the physical environment. The augmented reality device identifies optical features in the captured image data as anchor points. These anchor points are not expected to physically move in the physical environment. However, as the augmented reality device moves through the physical environment, the position of the anchor points will change in the frames of captured image data. The relative movement of the anchor points in successive frames of captured image data can be used to help determine a corresponding relative movement of the augmented reality device within the physical environment, so that an updated pose of the augmented reality device can be computed for a current frame. However, the estimated positions of the anchor points may be inaccurate, due to error in the sensor data itself and in the computations to recognize the optical features. As the estimated positions are recomputed at each frame, the anchor points may appear to drift due to these errors. When simulating large objects or objects that are located far away from the augmented reality device, even a small error in the position of an anchor point that is close to the augmented reality device can be compounded and produce a substantially large adverse visual effect in a portion of the object that is located a significant distance from the augmented reality device.